Skip to content

ARROW-11159: [Developer] Consolidate pull request related jobs#9120

Closed
kou wants to merge 1 commit into
apache:masterfrom
kou:ci-pr
Closed

ARROW-11159: [Developer] Consolidate pull request related jobs#9120
kou wants to merge 1 commit into
apache:masterfrom
kou:ci-pr

Conversation

@kou

@kou kou commented Jan 7, 2021

Copy link
Copy Markdown
Member

No description provided.

@kou

kou commented Jan 7, 2021

Copy link
Copy Markdown
Member Author

Demo: kou#10

@github-actions

github-actions Bot commented Jan 7, 2021

Copy link
Copy Markdown

@jorgecarleitao jorgecarleitao left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is no reason to have them in multiple lines when the status report is opened, so, LGTM.

I am curious as to why this addresses the slow queue. AFAIK all these jobs run in seconds.

@kou

kou commented Jan 7, 2021

Copy link
Copy Markdown
Member Author

This may not address the slow queue... We don't know without try...

@kou kou closed this in 3a7048a Jan 7, 2021
@kou kou deleted the ci-pr branch January 7, 2021 08:10
@jorgecarleitao

Copy link
Copy Markdown
Member

My hypothesis for a slow queue is that it is driven by cost: we simply have too many expensive (but unnecessary) jobs running.

  1. Every push triggers a 30m+ job (integration tests). Only about 5m of this job is the actual testing
  2. jobs are not canceled if there is a push after the push that triggered the job, even though those jobs are very likely not relevant once there is a new commit / rebase.
  3. there are jobs from language X running on very push to a user's repo, even when code for X does not change.

These 3 aspects very quickly fill any queue.

@kou

kou commented Jan 8, 2021

Copy link
Copy Markdown
Member Author
  1. It seems that we need to speed up integration test job. (Caching built binaries and reusing not changed binaries?)
  2. https://github.com/apache/arrow/graphs/commit-activity shows that we have about 10 pushes in apache/arrow master per day. If we can fix 1., this may not be a critical.
  3. It seems that we need auto cancel feature that exists on Travis CI and AppVeyor.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants